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1.  INTRODUCTION:  Narrative  that  briefly  (one  paragraph)  describes  the  subject,  purpose  and 
scope  of  the  research. 

Over  one  million  US  Veterans  have  visual  impairments  (VI)  that  affect  their  ability  to  perform 
activities  of  daily  living — a  number  that  continues  to  rise  as  the  US  Veteran  population  ages  [2], 
While  previous  research  has  explored  combining  mobile  cameras  and  computer  vision  to  support 
people  with  VI  for  at-a-distance  information  tasks  such  as  navigation  {e.g.,  [1,6,  16,  18,  31,  39]), 
facial  recognition  {e.g.,  [10,  23,  24]),  and  spatial  perception  {e.g.,  [19-21]),  they  fail  to  support 
proximal  information  accessed  through  touch.  In  our  research,  we  are  pursuing  a  fundamentally 
new  approach:  a  vision-augmented  touch  system  called  HandSight  aimed  at  supporting  activities  of 
daily  living  (ADLs)  by  sensing  and  feeding  back  non-tactile  information  about  the  physical  world 
as  it  is  touched.  HandSight  consists  of  tiny  CMOS  cameras  (lx  1mm  )  and  micro-haptic  actuators 
integrated  into  one  or  more  fingers,  computer  vision  and  machine  learning  algorithms  to  support 
fingertip-based  sensing,  and  a  smartwatch  for  processing,  power,  and  speech  output.  Since  touch  is 
a  highly  attuned  means  of  acquiring  information  for  people  with  VI  [12,  34],  we  hypothesize  that 
collocating  the  camera  with  the  touch  itself  will  enable  new  and  intuitive  assistive  applications. 
Imagine,  for  example,  touching  a  printed  page  and  receiving  speech  output  while  haptic  cues  guide 
your  fingers  along  the  text,  touching  a  piece  of  clothing  to  hear  suggestions  for  coordination,  or 
even  picking  up  a  bell  pepper  and  instantly  recognizing  that  it  is  red  or  green.  In  the  next  sections, 
we  articulate  the  key  objectives  of  our  work  and  our  progress  thus  far. 


2.  KEYWORDS:  Provide  a  brief  list  of  keywords  (limit  to  20  words). 


Blind,  visually  impaired,  wearable  computing,  computer  vision,  vision-augmented  touch,  activities 
of  daily  living  (ADLs),  assistive  technology,  haptics,  real-time  OCR 


3.  ACCOMPLISHMENTS:  The  PI  is  reminded  that  the  recipient  organization  is  required  to 
obtain  prior  written  approval  from  the  awarding  agency  Grants  Officer  whenever  there  are 
significant  changes  in  the  project  or  its  direction. 

What  were  the  major  goals  of  the  project? 

List  the  major  goals  of  the  project  as  stated  in  the  approved  SOW.  If  the  application  listed 
milestones/target  dates  for  important  activities  or  phases  of  the  project,  identify  these  dates  and 
show  actual  completion  dates  or  the  percentage  of  completion. 


We  have  two  high-level  goals  for  HandSight: 

1.  First,  to  develop  the  basic  building  blocks  of  an  extensible  HandSight  platform  that 
will  support  a  range  of  ADL  applications.  This  includes:  (i)  developing  computer  vision  and 
machine  learning  algorithms  to  support  basic  sensing  needs  from  a  finger-mounted  camera 
{e.g.,  optical  character  recognition,  surface  and  color  classification,  gesture  recognition); 
and  (ii)  investigating  physical  design  and  feedback  modalities  {e.g.,  augmenting  one  vs. 
both  index  fingers,  using  directional  haptic  feedback  to  guide  the  user’s  touch). 
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2.  Second,  to  explore  and  demonstrate  the  potential  of  HandSight  through  three  proof-of- 
eoneept  applieations:  reading,  dressing,  and  teehnology  aeeess.  Our  goal  to  demonstrate  the 
benefits  of  HandSight  both  for  blind  users  and  for  the  larger  VI  population.  The  VA 
estimates  that  157,000  US  Veterans  are  legally  blind,  and  more  than  one  million  have  low 
vision  [2],  Unobtrusively  reading  text,  for  example,  holds  signifieant  promise  for  both 
populations. 

In  the  first  year  of  our  funded  work,  we  have  focused  largely  on  the  first  high-level  goal — that  is, 
developing  the  basic  components  for  an  extensible  platform — including  designing  and  iterating  on 
physical  form  factors,  developing  computer  vision  algorithms  to  extract  attributes  of  the  physical 
world  and  to  support  on-body  interaction,  and  experimenting  with  haptic  feedback  options  to  guide 
the  user’s  finger/hand. 

Below,  we  list  the  tasks  from  our  SOW  and  approximate  completion  rates.  We  enumerate  what  was 
accomplished  for  these  tasks  in  the  next  section  (note:  if  a  task  is  not  listed  below,  then  no  work 
has  been  accomplished  in  that  area). 

•  Task  1:  Develop  responsive  haptic  feedback:  ~75%  complete 

•  Task  2:  Develop  physical  form  factor:  ~25%  complete 

•  Task  3:  Develop  sensing  algorithms  and  evaluate  using  controlled  datasets:  ~35%  complete 

•  Task  4:  Design,  build,  and  evaluate  a  reading  application:  ~30%  complete 

•  Task  6:  Design,  build,  and  evaluate  technology  access  application:  ~25%  complete 

What  was  accomplished  under  these  goals? 

For  this  reporting  period  describe:  1)  major  activities;  2)  specific  objectives;  3)  significant 
results  or  key  outcomes,  including  major  findings,  developments,  or  conclusions  (both  positive 
and  negative);  and/or  4)  other  achievements.  Include  a  discussion  of  stated  goals  not  met. 
Description  shall  include  pertinent  data  and  graphs  in  sufficient  detail  to  explain  any  significant 
results  achieved.  A  succinct  description  of  the  methodology  used  shall  be  provided.  As  the 
project  progresses  to  completion,  the  emphasis  in  reporting  in  this  section  should  shift  from 
reporting  activities  to  reporting  accomplishments. 

In  this  section,  we  report  on  our  accomplishments  for  Tasks  1-4  and  Task  6.  While  the  Tasks 
themselves  share  many  overlapping  pursuits  {e.g.,  the  physical  design  and  hardware  impacts  nearly 
every  Task),  we  attempt  to  describe  the  main  aspects  of  each  Task  separately. 

TASK  1:  DEVELOP  RESPONSIVE  HAPTIC  FEEDBACK:  -75%  COMPLETE 
For  many  ADL  tasks,  we  need  to  actively  guide  the  user’s  finger,  hand,  or  even  arm  toward  or 
along  some  object.  For  example,  in  the  reading  task  (Task  4),  the  user’s  finger  needs  to  be  guided 
along  lines  of  text  in  order  for  the  finger- worn  camera  to  scan  and  read  words.  Thus,  Task  1 
focuses  primarily  on  how  to  use  haptics  {e.g.,  vibration,  pressure,  etc.)  to  guide  the  user’s  finger 
and  hand  movement  to  accomplish  some  goal  {e.g.,  reading  printed  text).  Our  Task  1  progress  can 
be  broken  into  three  categories  of  work:  (i)  exploring  and  experimenting  with  haptic  hardware;  (ii) 
designing,  developing,  and  evaluating  finger-mounted  haptic  guidance;  and  (iii)  designing, 
developing,  and  evaluating  wrist-mounted  haptic  guidance.  While  our  SOW  describes  a  controlled 


lab  study  of  the  most  promising  haptic  feedback  options  at  the  Atlanta  VA  in  months  9-1 1,  we  have 
not  yet  conducted  this  study.  However,  we  meet  weekly  with  our  Atlanta  VA  co-PI,  David  Ross, 
have  sent  one  graduate  student  to  Atlanta  to  work  with  and  present  to  David  and  his  colleagues 
about  our  progress,  and  plan  to  run  this  user  study  by  the  end  of  2015. 


Task  1.1:  Exploring  and  Experimenting  with  Haptic  Hardware 

We  surveyed  14  commercial  haptic  hardware  options  informed  by  the  haptics’  research  literature 
and  our  own  online  searches.  We  purchased  and  informally  evaluated  seven  of  these  options 
including  piezoelectric  actuators  [37],  Dynalloy  Flexinol  actuator  wire  [9],  standard  vibrating 
motor  discs  (such  as  those  found  in  cell  phones)  [38,  45],  and  the  C-3  Factor  [4].  See  Figure  1  for 
some  example  haptic  actuators  we  have  explored.  Based  on  price,  size,  operating  characteristics, 
and  the  results  from  informal  experiments,  we  selected  the  vibrating  motor  discs  (far  left  in  Figure 
1)  for  our  initial  haptic  studies  but  we  will  continue  to  experiment  with  and  explore  other  options. 
As  noted  in  a  previous  report,  we  are  having  some  trouble  finding  haptic  solutions  that  are  as  small 
as  our  camera  hardware,  which  consequently  increases  the  size  of  the  entire  HandSight  prototype. 


Cheap  Motor 
lOmm 
$2 


Precision  Microdrives  LRA 
10mm 
$10 


Piezoelectric  Actuator 
2x3xl0mm 
$33  +  $100  driver 


C-3  Tactor 
20mm 
$340 


Figure  1:  Four  example  haptic  actuators  we  have  purchased  and  examined  in  our  lab. 


Task  1.2:  Designing,  Developing,  and  Evaluating  Finger-Mounted  Haptics 

To  help  us  study  haptic  options  in  detail,  we  developed  an  iPad-based  test  apparatus.  Here, 
blind/low-vision  participants  can  complete  finger-guidance  tasks — e.g.,  trace  a  line,  trace  a  shape, 
follow  lines  of  text — and  we  can  record  precise  finger  movements  (e.g.,  finger  position,  velocity, 
acceleration  and  proximity  to  the  projected  shape).  A  thin  sheet  of  paper  covers  the  iPad  to 
simulate  the  feeling  of  paper  (this  does  not  affect  the  capacitive  finger-location  sensing  of  the 
iPad).  See  Figure  2. 


T'mily,  he  smiled,  shrieked,  pounded  the 
ground,  and  l0v)kcu  ?y^ber  of 

the  family  to  the  next.  Still  smiling  and 
shrieking,  Nim  wenn^round  hugging  each 
member  6^  the  fa»^dy  Hp  nijiypH  with  and 
groomed  each  member  of  the  family  for 
almost  an  hour  before  thp  family  had  to 
leave.  People  who  were  famiiiar  with  flim’e 

Figure  2:  (left)  The  iPad  testbed  setup  for  our  lab  studies,  (right)  The  precise  finger  traces  collected  by  our  custom  iPad  test 
application  (green/red  lines  are  actual  finger  traces  from  a  participant  in  the  TACCESS  study;  green=on  line  so  no  feedback 

provided  and  red=feedback  provided). 
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Finger-mounted  haptic  prototype.  Our  finger-mounted  haptic  prototype  consists  of  two  disc 
vibration  motors  (8mm  diameter,  3.4mm  thick)  controlled  by  an  Arduino  Pro  Micro  that 
communicates  with  the  iPad  via  Bluetooth.  The  motors  are  attached  to  the  user’s  right  index  finger 
with  separate  Velcro  rings  (Figure  3),  one  on  top  of  the  finger  on  the  intermediate  phalange  and 
one  below  the  finger  on  the  proximal  phalange.  The  lower  motor  indicates  that  the  finger  should 
move  downward  and  the  upper  motor  indicates  the  opposite.  Neither  motor  vibrates  while  the 
user’s  finger  is  directly  over  the  current  line  of  text.  Vibration  intensities  off  the  line  range  from  a 
minimum  perceptible  strength  to  the  maximum  strength  the  motors  can  provide,  using  distance 
thresholds.  The  choice  to  vary  the  position  and  intensity  of  vibration  to  indicate  direction  and 
distance  was  also  motivated  by  our  prior  work  and  validated  in  pilot  sessions.  We  conducted  three 
studies  of  this  general  approach  published  across  three  venues — all  in  the  context  of  a  finger-reader 
application  (Task  4  in  our  SOW).  We  summarize  findings  below  and  ask  that  you  consult  the 
specific  publications  for  details. 


Haptic  Motors 


Figure  3:  Close-up  view  of  the  haptic  motor  configuration  used  in  our  TACCESS  research.  They  are  both  mounted  on  the  finger  via 
Velcro  rings.  The  top  motor  vibrates  when  the  user’s  finger  moves  below  the  line,  providing  upward  guidance;  the  bottom  motor 
vibrates  when  the  user’s  finger  moves  above  the  line,  providing  downward  guidance.  The  intensity  of  vibration  depends  upon  the 
distance  to  the  line,  achieving  maximum  intensity  at  127  pixels  (~2.4  cm). 

Initial  study  (ACVR’14  [43]).  We  ran  an  initial  study  with  four  blind  participants  comparing  an 
initial  finger-based  haptic  feedback  prototype  to  pitch-controlled  audio  with  our  iPad  testbed.  In 
short,  we  found  that  three  participants  preferred  the  audio  condition  while  one  preferred  combined 
audio+haptics.  While  preliminary,  these  findings  suggest  that  an  audio  channel  should  be 
considered  in  addition  to  a  haptic  guidance  channel  for  precise  guidance  tasks  such  as  line-by-line 
reading  with  a  finger.  These  results  are  published  at  ACVR’14  [43]. 

Follow-up  studies  (ASSETS’15  [7]),  iu  subuiissiou  to  TACCESS  [42]).  In  follow-up  work 
currently  in  submission  to  ACM  Transactions  on  Accessible  Computing  (TACCESS)  [42],  the  top 
accessibility  journal  in  Human-Computer  Interaction,  we  conducted  two  lab  studies  with  blind 
persons:  (i)  a  comparison  of  audio  and  haptic  directional  finger  guidance  with  20  blind  participants 
using  an  updated  version  of  our  iPad-based  testbed  and  a  refined  version  of  our  haptic  setup,  and 
(ii)  a  smaller  proof-of-concept  study  with  4  blind  participants  using  a  preliminary  wearable 
prototype  with  physical  paper.  The  primary  goal  of  the  first  study  was  to  compare  the  effects  of  the 
two  guidance  methods  in  terms  of  line  tracing  accuracy,  reading  speed,  comprehension,  and 
subjective  response.  The  second  study  explored  the  experience  of  using  HandSight  to  read  printed 
documents  and  compared  it  with  a  popular  mobile  application,  KNFB  Reader  iOS. 
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In  the  first  study,  we  found  similar  performance  between  haptic  and  audio  directional  guidance, 
although  audio  offered  a  slight  accuracy  advantage  for  line  tracing.  Subjective  feedback  also 
highlighted  tradeoffs  between  the  two  types  of  guidance,  such  as  the  interference  of  audio  guidance 
with  speech  output  and  the  potential  for  desensitization  to  haptic  guidance.  While  several 
participants  appreciated  the  direct  access  to  layout  information  provided  by  the  finger-based 
reading  approach,  important  concerns  also  arose  about  ease  of  use  and  the  amount  of  concentration 
required  to  read  in  this  way.  See  the  paper  for  details. 

Task  1.3:  Designing,  Developing,  and  Evaluating  Wrist-Mounted  Haptics 

In  addition  to  the  finger-mounted  haptic  approach  described  above,  we  have  also  designed, 
developed,  and  evaluated  a  wrist-mounted  haptic  approach.  As  our  envisionment  for  HandSight 
includes  a  smartwatch  for  processing  and  power,  we  wanted  to  explore  how  the  wristband  itself 
could  be  used  to  provide  useful  haptic  feedback  to  the  user.  The  wrist  offers  a  balance  between 
proximity,  sensitivity  [5],  surface  area,  and  social  acceptability.  The  larger  surface  area,  for 
example,  allows  for  a  greater  number  of  vibration  sources  than  the  finger.  Similar  to  the  finger- 
mounted  haptics  research,  we  pursued  a  human-centered,  iterative  design  approach  starting  with 
ideation  and  the  informal  exploration  of  a  number  of  initial  physical  prototypes,  followed  by  a 
series  of  iterative  refinements,  then  more  formal  experiments  with  externally  recruited  participants. 
The  physical  designs  are  described  in  the  Task  2  section. 

We  are  currently  exploring  physical  form  factors,  number  and  type  of  embedded  wristband  haptic 
devices,  and  haptic  modulation  approaches  {e.g.,  pulse,  continuous,  frequency,  etc.).  Thus  far,  we 
have  investigated  three  types  of  tasks:  (i)  direction,  (ii)  target  finding,  and  (iii)  and  route  tracing. 

To  examine  these  tasks  experimentally,  we  created  another  testbed  similar  to  that  used  in  Task 
1.2 — this  time,  however,  on  an  Android  tablet  rather  than  an  iPad  (Android  screenshots  are  in 
Figure  4).  While  we  have  tested  each  of  these  tasks  in  our  lab  with  pilot  participants,  we  have 
conducted  a  formal  experiment  with  only  the  first  task  (direction),  which  we  describe  below.  Other 
tasks  will  be  examined  soon  with  VI  users  at  the  Atlanta  VA  led  by  co-PI  David  Ross. 
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Training  task  Target  finding  task  Route  tracing  task 

Figure  4:  Two  of  the  three  experimental  tasks  in  our  Android  application:  the  (ii)  target  finding  task  and  the  (iii)  route  tracing  task. 

The  visuals  are  for  illustration  only  and  would  not  be  seen  by  participants.  The  yellow  line  in  the  target-finding  and  route-tracing  tasks 
shows  the  finger  trace  so  far,  while  the  green  line  (up  or  down)  indicates  which  motor  is  currently  vibrating  on  the  user’s  wrist. 


Initial  study  (in  submission  to  CHI’ 16).  In  this  work,  we  performed  a  controlled  lab  experiment 
to  evaluate  the  impact  of  wrist-based  vibro-motor  feedback  on  directional  hand  movement, 
comparing  lower-fidelity  (4-motor)  and  higher-fidelity  (8-motor)  wristbands.  Twenty  sighted 
participants  completed  a  series  of  non-visual  trials  consisting  of  interpreting  a  haptic  stimuli  and 
executing  a  2D  directional  movement  on  a  touchscreen.  We  assessed  movement  error  and  trial 
speed,  but  also  conducted  a  deeper  analysis  of  the  impact  of  specific  directions  on  performance. 
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The  haptic  feedback  worked  as  follows:  vibration  frequency  was  controlled  by  pulse-width- 
modulation  (PWM)  from  the  Arduino;  higher  voltage  corresponded  to  higher  vibration  frequency. 
The  frequency  range  used  for  the  experiment  was  62-156  Hz,  which  we  determined  through  two 
mechanisms.  First,  to  ensure  that  we  accurately  used  voltage  to  manipulate  vibration  frequency,  we 
evaluated  the  relationship  between  voltage  and  frequency  when  the  motor  is  worn  on  the  wrist.  As 
shown  in  Figure  5d,  this  relationship  was  roughly  linear  above  1.2V,  up  to  a  maximum  frequency 
of  156Hz  at  3V.  Second,  to  ensure  that  the  full  frequency  range  was  perceptible  by  users,  we 
conducted  a  simple  perception  test  with  6  participants  from  our  lab.  We  increased  the  frequency 
continuously  from  0  Hz  until  the  participant  reported  a  light  vibration,  then  decreased  it  from  156 
Hz  until  no  vibration  was  felt.  To  be  conservative,  we  selected  the  maximum  value  among  all 
reported  thresholds  (62  Hz)  as  a  perceptible  lower  bound  on  our  vibration  range. 


To  indicate  a  target  direction,  one  or  two  motors  vibrated.  If  the  direction  coincided  exactly  with  a 
motor’s  placement,  that  single  motor  vibrated  at  maximum  frequency  (156  Hz).  If  the  direction  was 
between  two  motors,  those  motors  vibrated  with  linearly  interpolated  frequencies  within  the  62- 
156  Hz  range.  For  example,  45°  is  exactly  between  two  motors  on  the  4-motor  wristband,  so  those 
motors  vibrated  at  the  same  frequency,  while  for  70°,  one  motor  vibrated  at  higher  frequency  than 
the  other.  More  formally: 


fl  —  (^2  ^  ^  start) 

/z  =  ~  ^1  ~  ^start) 


fmax  fmi 

^2  ~  ^1  ~  ^ start 

fmax  ~  fmin 
*  - 

Oy  01  0  Qtnrf 


'  fmi'i 
+  fm 


where  =  156,  =  62,  ^^and  02  indicate  the  motor  placements  and  0  is  the  target 

direction  (02  >  0  >  0i).  To  use  the  full  range  of  perceptible  frequencies,  the  second  motor  did  not 
begin  vibrating  until  0 start  ^  11.25°  past  0^. 


The  contributions  of  this  study  include:  (i)  empirical  evidence  that  doubling  the  number  of  haptic 
motors  reduces  directional  movement  error  and  that  movement  error  is  greater  to  the  upper-left 
than  in  other  directions;  (ii)  identification  of  a  maximum  threshold  of  -25°  accuracy  using  our 
approach;  (iii)  design  considerations  for  incorporating  directional  haptic  guidance  into  a 
smartwatch  band.  A  description  of  the  hardware  and  physical  design  is  provided  in  Task  2.1  below. 


TASK  2:  DEVELOP  PHYSICAL  FORM  FACTOR:  -25%  COMPLETE 

At  a  high-level,  the  physical  design  of  HandSight  can  be  broken  down  into  four  categories  of  work: 
(i)  identifying  and  experimenting  with  camera,  haptic,  smartwatch,  and  wristband  hardware,  (ii) 
exploring  where  and  how  to  mount  the  camera  and  haptics  (e.g.,  rings,  glovces)  and  trying  to 
understand  the  advantages/disadvantages  of  each  placement  on  algorithm  performance  and  user 
comfort,  (iii)  exploring  more  holistic  aspects  of  the  design  such  as  wiring,  power,  etc.,  (iv) 
exploring  the  aforementioned  usability  aspects.  The  physical  design  and  form  factor  of  HandSight 
impacts  everything  from  user  comfort  to  the  design  and  operation  of  our  algorithms.  For  example, 
a  finger-mounted  camera  on  the  proximal  phalange  will  have  a  larger  field-of-view  than  one 
mounted  on  the  intermediate  or  distal  phalange  for  up-close  interactions — ^this  would  allow  the 
camera  to  see  more  information  and,  perhaps,  make  object/text  recognition  easier.  Similarly,  in 
investigating  camera  hardware  and  lenses,  we  are  interested  in  optimizing  for  size,  power  usage, 
and  resolution  with  a  large  field-of-view,  adaptive  focus,  and  self-illumination.  Higher  resolution. 
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self-illumination,  and  an  adaptive  focus  eases  many  computer  vision  problems  such  as  dealing  with 
shadowing,  image  artifacts  due  to  low  resolution  capture,  etc.  Thus  far,  our  focus  has  been 
primarily  on  creating /wnchona/  form  factors  {i.e.,  categories  (i)  and  (ii))  rather  than  attractive, 
robust,  or  aesthetic  form  factors.  Our  focus  will  shift  to  these  latter  attributes  once  we  have 
determined  a  promising  functional  design. 

Task  2.1  Finding  and  Experimenting  with  Hardware 

Below,  we  describe  the  hardware  related  to  the  camera,  haptics,  audio,  and  smartwatch. 

Camera  hardware.  HandSight’s  primary  sensing  hardware  consists  of  one  or  more  finger- 
mounted  cameras.  The  current  single-camera  prototype  uses  a  1  x  Imm^  A  WAIBA  NanEye  2C 
camera  [33]  that  can  capture  250x250  resolution  images  at  44  frames  per  second  (fps).  The 
NanEye  was  originally  developed  for  minimally  invasive  surgical  procedures  such  as  endoscopies 
and  laparoscopies  and  is  thus  robust,  lightweight,  and  precise.  We  have  also  explored  a  NanEye 
camera  with  four  LEDs  coincident  with  the  lens,  which  allows  for  increased  illumination  control. 
This  is  the  camera  used  in  both  our  ACVR’  14  and  TACCESS  work.  More  recently,  we  have 
experimented  with  a  stereo  NanEye  version,  which  is  larger,  but  may  be  able  to  provide  depth  and 
proximity  information  (2x44fps,  1x2. 2mm  )  and  a  faster,  higher  resolution  NanEye  eamera 
(lOO^s,  640x640,  3.4x3.4mm^),  whieh  again  though  larger,  may  allow  us  to  more  aeeurately 
eompute  optieal  flow  during  finger  movement  (as  indieated  by  previous  work  [47]). 

While  the  NanEye  eameras  have  reeeived  most  of  our  attention,  we  have  also  explored  other 
options.  Below,  we  list  the  full  range  of  eameras  we  have  investigated  via  informal  testing  in  our 
lab  along  with  pros/eons.  The  infrared  (IR)  eameras  listed  are  for  future  work  related  to  the  on- 
hand  loealization  researeh  deseribed  in  Task  3.2  (speeifieally,  they  are  used  for  vein  deteetion). 

•  NanEyes:  http://www.cmosis.com/products/product  detail/naneye  module 

o  NanEye  2C,  NanEye  GS,  and  NanEye  GS  Stereo 

o  Extremely  tiny  with  good  image  quality  &  focal  length;  high  frame  rate  &  programmable 
o  Very  expensive 

•  Microlens:  http://www.amazon.eom/gp/product/B009N8JE9O/ 

o  Portable  Microscope  used  in  CVPR’  1 6  research 
o  Adjustable  focal  length  (as  close  as  1cm),  good  image  quality 
o  Very  large,  requires  manual  LED  and  focal  length  adjustments 

•  Endoscope/Boroscope:  http://www.amazon.com/Newest-Endoscope-Borescope-Insepction- 
Camera/dp/BQQMNBKFHC/ 

o  Cheap  and  durable 

o  Low  frame  rate,  bad  image  quality,  fixed  foeus  (infinite) 

•  Sony  Playstation  Eye:  http://www.amazon.eom/Sonv-PlavStation-Camera-Bulk-Paekaging-Pe/dp/B0072I2240 

o  Sony  Playstation  Eye  camera  with  IR  filter  removed  and  visible  light  filter  added 
o  For  vein  detection 

o  Very  cheap,  easily  hackable 

o  Small  for  a  webeam,  but  too  large  to  be  wearable  on  the  finger 

•  Kinect:  http://www.xbox.eom/en-US/xbox-360/aeeessories/kineet 

o  Mierosoft  Kinect  for  Windows  vl 

o  Good  quality  IR  images  for  vein  detection  (includes  850nm  filter) 
o  Programmable,  easily  switehes  between  IR  and  RGB 
o  Large,  low  resolution,  and  fixed  foeal  length  >  1ft 

•  Raspberry  PI  NoIR:  https://www.raspberrvpi.org/produets/pi-noir-camera/ 

o  Raspberry  Pi  NoIR  eamera  with  external  visible  light  filters 
o  Good  image  quality,  manually  adjustable  focal  length,  small,  programmable 
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Figure  5:  (a)  The  four-  and  eight-motor  haptic  wristband  prototypes,  (b)  A  close-up  of  our  haptic  design;  motors  were  placed  in  the 
3D-printed  cases  using  magnetic  attachments,  and  the  wristband  faced  downward  so  that  the  motors  directly  contacted  the  skin,  (c) 
Our  custom-designed  wristband  could  accommodate  wrists  of  various  shapes  and  sizes.  Motor  placement  was  adjusted  per  participant 
to  match  this  figure,  (d)  A  voltage  input  vs.  vibration  frequency  response  graph  as  measured  using  a  piezoelectric  sensor  while  the 
motor  was  worn  on  a  wrist.  Results  were  used  to  verify  a  linear  relationship  between  voltage  input  and  frequency  and  to  empirically 
determine  minimum  voltage. 

Wristband  Haptics  (in  submission  to  CHI’16).  Our  initial  hardware  implementation  for  finger- 
mounted  hapties  is  deseribed  in  Tasks  1.1  and  1.2  above.  Here,  we  deseribe  the  four-vibromotor 
and  eight-vibromotor  wristbands  used  in  our  first  wrist-based  hapties  study  (in  submission  to 
CHI’ 16).  The  wristbands,  shown  in  Figure  5a,  were  identieal  exeept  for  the  number  of  motors  (four 
or  eight).  We  used  eireular  dise  vibro-motors  10  mm  in  diameter,  with  maximum  voltage  of  5V  and 
maximum  frequeney  of  183Hz.*  These  motors  are  inexpensive  and  ubiquitous,  and  the  flat  design 
means  they  ean  be  easily  integrated  into  a  future  envisioned  smartwateh  wristband.  Our  eustom 
design  addresses  three  issues:  vibration  transfer  along  the  band,  variation  in  wrist  size,  and  the  non- 
uniform  shape  of  a  wrist.  To  effeetively  isolate  the  motors  and  limit  vibration  transfer — an  issue  in 
our  early  designs — we  mounted  the  motors  on  a  band  separate  from  the  wiring  and  housed  them 
magnetieally  in  3D-printed  eases  eonneeted  only  by  thin  elastie  thread  (Figure  5b).  The  band  with 
the  wiring  in  turn  eonneeted  to  the  Arduino  for  eommunieation  and  power. 

Beeause  the  wrist  is  not  a  uniform  oval  (Figure  5e),  plaeing  the  motors  equidistantly  around  the 
band  (as  in  [41,  46])  means  that  the  motors  are  not  neeessarily  at  the  position  the  user  expeets — for 
example,  the  right-most  loeation  on  the  wrist  may  not  be  midway  between  the  up  and  down 
positions.  To  address  variation  in  wrist  sizes  and  shapes,  our  prototype  is  adjustable.  The  band  with 
the  motors  is  threaded  through  the  motor  eases  rather  than  affixed,  whieh  allows  the  eases  to  slide 
(with  effort)  along  the  band,  and  allows  the  band  to  be  tightened  or  loosened  (based  on  [17]).  Thus, 
the  wristband  eould  be  easily  reeonfigured  to  support  a  wide  range  of  wrist  sizes. 

Audio.  We  have  used  audio  to  both  eomplement  the  haptie  ehannel  (e.g.,  notifieations)  as  well  as 
to  serve  as  a  baseline  eondition  for  some  experiments.  Currently  audio  signals  are  eonveyed  via  a 
tablet  or  laptop  but  will  be  ineorporated  direetly  into  future  smartwateh  designs. 

Smartwateh.  For  the  smartwateh,  we  have  largely  relied  on  our  own  prototype  substitute — a  small 
Arduino  board  with  eustom  3D  printed  ease  and  wristband.  We  expeet  to  replaee  this  with  an 
Android-based  smartwateh  in  the  future. 

TASK  3:  DEVELOPING  &  EVALUATING  SENSING  ALGORITHMS:  -35%  COMPLETE 
As  previously  noted,  we  have  two  main  interrelated  objeetives  for  HandSight:  first,  to  develop  the 
basie  building  bloeks  of  an  extensible  platform  and  seeond,  to  explore  and  demonstrate  the 
potential  of  HandSight  through  three  proof-of-eoneept  applieations  in  reading,  dressing,  and 
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technology  access.  Below,  we  describe  our  progress  thus  far  in  developing  and  evaluating  our 
camera-based  sensing  and  inference  algorithms  for  the  latter  two  of  these  application  areas:  reading 
and  technology  access. 

Task  3.1  Developing  Real-time  OCR  with  Finger-Mounted  Camera 

While  our  long-term  plan  is  to  integrate  real-time  OCR  processing  into  a  smartwatch,  the  algorithm 
presented  below  was  run  on  a  laptop  connected  directly  to  the  finger-mounted  camera.  Our  current 
HandSight  implementation  involves  a  series  of  frame-level  processing  stages  followed  by  multi- 
frame  merging  once  the  complete  word  has  been  observed.  Below,  we  summarize  our  five-stage 
OCR  process  and  some  preliminary  experiments  evaluating  performance.  For  more  detail,  see  our 
ACVR’14  paper. 

Stage  1:  Preprocessing.  We  acquire  grayscale  video  frames  at  ~40ips  and  250x250px  resolution 
from  the  NanEye  camera  (Figure  6).  With  each  video  frame,  we  apply  four  preprocessing 
algorithms  to  correct  radial  and  (slight)  tangential  distortion,  to  control  lighting  for  the  next  frame, 
to  reduce  noise,  and  to  reduce  false  positives. 

Stage  2:  Perspective  and  Rotation  Correction.  To  correct  perspective  and  rotation  effects,  we 
apply  an  efficient  approach  that  relies  on  the  parallel  line  structure  of  text  for  rectification. 


Stage  3:  Text  Detection.  The  goal  of  the  text  detection  stage  is  to  build  up  a  hierarchy  of  text 
lines,  words,  and  characters.  We  split  the  image  into  lines  of  text  by  counting  the  number  of  text 
pixels  in  each  row  and  searching  for  large  gaps,  then  segment  lines  into  words  using  a  similar 
process,  and  finally  segment  each  word  into  individual  characters  by  searching  for  local  minima  in 
the  number  of  text  pixels  within  each  column. 

Stage  4:  Character  Classification.  Real-time  performance  is  important  for  responsive  feedback, 
which  prevents  us  from  using  established  OCR  engines  such  as  Tesseract.  Thus,  we  compute 
efficient  character  features  and  perform  classification  using  a  support  vector  machine  (SVM). 

Stage  5:  Tracking  and  final  OCR  result  output.  The  camera’s  limited  field  of  view  means  that  a 
complete  word  is  seldom  fully  within  a  given  frame.  We  must  track  the  characters  between  frames 
and  wait  for  the  end  of  the  word  to  become  visible  before  we  can  confidently  identify  it.  Character 
tracking  uses  sparse  low-level  features  for  efficiency.  First,  we  extract  FAST  comers  [40],  and 
apply  a  KLT  tracker  [44]  at  their  locations.  We  estimate  the  homography  relating  the  matched 
comers  using  the  random  sample  consensus  [8].  After  determining  the  motion  between  frames,  we 
relate  the  lines,  words,  and  individual  characters  by  projecting  their  locations  in  the  previous  frame 
to  the  current  frame  using  the  computed  homographies.  The  bounding  boxes  with  the  greatest 
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amount  of  overlap  after  projection  determine  the  matches.  When  the  end  of  a  word  is  visible,  we 
sort  the  aggregated  character  classifications  and  accept  the  most  frequent  classification.  This 
process  can  be  improved  by  incorporating  a  language  dictionary  model,  albeit  at  the  expense  of 
efficiency.  A  text-to-speech  engine  reads  back  the  identified  word. 

Algorithm  Evaluation.  We  investigated  the  effect  of  camera  perspective  and  rotation  on  OCR 
accuracy  as  well  as  the  effect  of  finger  movement  speed  on  OCR  accuracy.  For  the  former,  we 
created  an  artificially  rotated  and  distorted  our  images  and  tested  the  resulting  performance  of  our 
algorithms  (Figure  7a  and  b).  For  the  latter,  we  recorded  five  different  speeds  using  a  single  line  of 
text.  The  results  are  presented  in  Figure  7c.  With  greater  speed,  motion  blur  is  introduced,  and 
feature  tracking  becomes  less  accurate.  In  our  experience,  a  “natural”  finger  speed  movement  for 
sighted  readers  is  roughly  2-3cm/s.  So,  with  the  current  prototype,  one  must  move  slower  than 
natural  for  good  performance.  We  plan  on  compensating  for  this  effect  in  the  future  using  image 
stabilization  and  motion  blur  removal,  as  well  as  incorporating  a  higher  frame  rate  camera 
(lOOfps).  Other  future  work  is  described  later  in  the  ‘plans  for  next  reporting  period’  section. 


15 


•  Before 


A  %  ^  ^ 


-45  -30  -15  0  15  30  45 

Lateral  Perspective  Angle  (°) 

(a)  Performance  of  lateral  perspective 
and  rotation  rectification  algorithm. 
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(b)  Effect  of  lateral  perspective  angle  on 
accuracy  (before  and  after  correction). 
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(c)  Effect  of  finger  speed  on 
character-  and  word-level  accuracy. 


Figure  7:  Results  from  preliminary  evaluations  of  our  (a-b)  Stage  2  algorithms  and  (c)  the  effect  of  finger  speed  on  overall 
character-  and  word-level  accuracy. 


Task  3.1  Developing  On-Hand  Localization  Algorithms 

While  we  primarily  envision  HandSight  as  enabling  access  to  the  physical  world,  we  are  also 
exploring  another  use  case:  increasing  mobile  technology  access  through  on-body  interaction. 
Mobile  devices  have  become  critical  to  everyday  tasks  from  communication  to  navigation,  yet 
persistent  accessibility  challenges  remain  [48].  As  an  alternative  to  the  touchscreen,  HandSight  will 
allow  users  to  control  the  screenreader  software  on  their  mobile  device  by  making  taps  or  swipes 
on  their  own  hands.  Users  who  do  not  need  a  visual  display  will  thus  not  need  to  pull  the  device  out 
of  their  pocket  or  bag.  On-hand  gestures  should  also  offer  increased  tactile  and  proprioceptive 
feedback  compared  to  the  smooth  surface  of  the  touchscreen.  While  arbitrary  surface  input  has 
been  previously  proposed  [3,  13-15,  29,  32,  47],  its  potential  to  improve  technology  access  has 
been  overwhelmingly  ignored  (with  one  small  exception  [14]). 

We  aim  to  investigate  two  specific  on-hand  interactions  to  increase  mobile  accessibility:  (i) 
detecting  and  localizing  on-hand  taps  (e.g.,  a  blind  user  tapping  on  a  specific  finger  tip  to  invoke  an 
action  on  their  mobile  device)  and  (i)  detecting  gestures  (e.g,  a  blind  user  swiping  across  a  specific 
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location  on  their  palm  to  silence  an  alarm).  To  our  knowledge,  such  localization  has  never  been 
done  with  finger-mounted  cameras.  We  describe  work  below  that  will  be  submitted  to  CVPR’  16  in 
early  November  (CVPR  is  one  of  the  premiere  publication  venues  in  computer  vision). 

Initial  On-Hand  Interaction  Algorithm  Work  (to  be  submitted  to  CVPR’ 16  in  November). 

Since  the  pahn  and  fingers  offer  a  high  density  of  discriminable  features  that  are  well  studied  in  the 
context  of  biometrics  [22,  25,  49],  they  are  a  promising  surface  for  image-based  localization. 
Proprioceptive  sensitivity  has  also  been  shown  to  be  greatest  on  the  tips  of  the  fingers  and  the  palm 
of  the  hand  [11],  making  them  an  appealing  choice  for  a  touch  surface.  However,  other  locations 
may  also  work  well;  in  this  work  we  explore  several  different  regions,  including  the  fingers  and 
palm  as  well  as  fingernails,  knuckles,  wrist,  and  back  of  the  hand. 

Our  approach  extends  [47]  for  on-body  input,  using  a  hierarchical  texture  classifier  to  estimate  the 
approximate  touch  region  on  the  body  (see  Table  2),  given  a  close-up  images  of  the  surface.  We 
then  refine  the  location  estimate  using  keypoint  matching  and  geometric  verification.  Although  we 
focus  primarily  on  the  hand  as  an  input  surface  due  to  its  size,  sensitivity,  and  visually 
discriminable  features,  our  approach  is  flexible  enough  to  adapt  to  other  body  locations  or  general 
surfaces  as  well.  We  evaluate  our  classification  algorithms  on  a  dataset  consisting  of  30  individuals 
and  17  different  close-up  regions  on  the  hand  (Figure  8;  10,198  micro-lens  images  in  all), 
achieving  a  preliminary  average  classification  accuracy  of  96.4%  using  n-fold  cross  validation. 
More  detailed  results  are  shown  in  Tables  1-3. 


Figure  8:  Seventeen  locations  of  images  being  collected  and  classified  for  the  on-hand  localization  algorithmic  work. 


Algorithmic  Robustness 


Variable 

Conditions 

Average  Accuracy 
Across  17  Locations 

Rotation 

15°  increments 

99.7%  {SD=QA%) 

Translation 

25%-100%  overlap 

98.2%  (SD=23%) 

Scale 

0.5x-2x 

78.9%  (SD=21.5%) 

Lighting 

Overhead/Left/Ri^ht 

52.5%  (SD  N/A) 

Pose 

Extended/Relaxed 

58.0%  (SD  N/A) 

Group-Level  Confusion  Matrix 


Palm 

Finger 

Nail 

Knuckle 

Other 

Palm 

99.0% 

0.5% 

0.5% 

Finger 

0.6% 

99.3% 

0.1% 

Nail 

0.2% 

0.1% 

99.7% 

0.1% 

Knuckle 

0.2% 

99.1% 

0.7% 

Other 

0.6% 

0.1% 

0.5% 

98.8% 

Table  1:  Results  of  a  series  of  algorithmic  robustness  Table  2:  Classification  accuracy  for  classes  at  the  group  level, 

experiments  using  a  synthetic  database  of  99000  images  with  Each  cell  indicates  the  percentage  of  images  assigned  to  a 

known  parameters.  predicted  class  (column)  for  each  actual  class  (row). 
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Palm 

Finders 

Nails 

Knuckles 

Other 

C 

U 

D 

L 

R 

pt 

2nd 

^rd 

^th 

5* 

pt 

2nd 

pt 

2nd 

BH 

ow 

IW 

Palm  Center 

98.3% 

0.2% 

0.2% 

0.2% 

0.2% 

0.5% 

0.5% 

Palm  Up 

0.2% 

98.5% 

0.2% 

0.2% 

0.2% 

0.3% 

0.2% 

0.3% 

Palm  Down 

0.3% 

1.2% 

95.7% 

0.2% 

1.7% 

0.3% 

0.2% 

0.2% 

0.2% 

0.2% 

Palm  Left 

0.3% 

0.3% 

0.2% 

98.7% 

0.3% 

0.3% 

0.7% 

Palm  Ri^ht 

0.7% 

0.5% 

0.3% 

0.5% 

97.5% 

0.2% 

0.2% 

0.2% 

Finder 

0.5% 

0.2% 

0.2% 

0.7% 

96.3% 

0.3% 

0.5% 

0.5% 

0.7% 

0.2% 

2^^  Finder 

0.3% 

0.2% 

0.3% 

95.8% 

1.7% 

0.5% 

1.2% 

3^^  Finder 

0.3% 

0.2% 

1.3% 

95.4% 

2.2% 

0.7% 

Finder 

0.2% 

0.3% 

1.8% 

95.3% 

2.3% 

5^^  Finder 

0.2% 

0.2% 

0.3% 

0.5% 

1.5% 

97.0% 

0.3% 

Nail 

0.2% 

98.2% 

1.7% 

2"'^  Nail 

02% 

0.2% 

0.5% 

99.0% 

0.2% 

Knuckle 

0.2% 

97.3% 

1.2% 

0.2% 

0.2% 

1.0% 

2”^  Knuckle 

0.2% 

0.8% 

98.8% 

0.2% 

Back  of  Hand 

0.2% 

0.5% 

0.2% 

92.2% 

4.7% 

2.3% 

Outer  Wrist 

0.2% 

0.2% 

6.0% 

90.2% 

3.5% 

Inner  Wrist 

0.7% 

0.7% 

0.2% 

0.2% 

0.3% 

0.2% 

0.7% 

0.2% 

0.8% 

0.5% 

96.2% 

Table  3:  Classification  accuracy  for  classes  at  the  region  level,  averaged  across  20  trials  and  30  participants.  Each  cell  indicates  the 
percentage  of  images  assigned  to  a  predicted  class  (column)  for  each  actual  class  (row). 

The  contributions  of  this  work  include:  (i)  robust  algorithms  for  recognizing  several  different 
regions  of  the  hand,  which  will  enable  on-body  interactions  using  a  wearable  finger-mounted 
camera;  (ii)  classification  results  for  a  preliminary  dataset  consisting  of  30  individuals,  achieving 
96.4%  average  accuracy;  (iii)  analysis  of  palm  distinctiveness  and  similarities  between  users, 
which  may  impact  accuracy  and  scalability.  See  paper  for  details. 

TASK  4:  DESIGN,  BUILD,  &  EVALUATE  READING  APP:  -30%  COMPLETE 

The  progress  on  Task  4  has  largely  been  covered  by  the  above  sections.  In  summary,  while  we 
have  built  an  initial  real-time  finger-reading  hardware  prototype,  most  of  work  has  focused  on 
guiding  the  user’s  fmger/hand  along  a  page  of  text  (to  appropriately  scan  the  words).  In  our 
ACVR’  14  paper  [43],  we  present  an  initial  HandSight  hardware  prototype  with  a  single  finger- 
mounted  camera  and  two  haptic  rings  that  provide  continuous  line-by-line  guidance.  The  finger 
camera-based  OCR  algorithms  were  evaluated  experimentally  with  synthetically  derived  data  while 
the  system  itself  was  evaluated  in  an  initial  user  study  with  four  VI  users.  We  conducted  a  follow¬ 
up  study  (in  submission  to  TACCESS  [42])  with  19  blind  participants,  comparing  audio  and  haptic 
directional  finger  guidance  within  an  iPad-based  testbed.  We  later  asked  four  of  those  participants 
to  return  and  provide  feedback  on  a  slightly  updated  wearable  prototype  from  the  ACVR  work. 
Findings  from  the  controlled  experiment  show  similar  performance  between  haptic  and  audio 
directional  guidance,  although  audio  may  offer  an  accuracy  advantage  for  tracing  lines  of  text. 
While  several  participants  appreciated  the  direct  access  to  layout  information  provided  by  finger- 
based  exploration  of  text,  important  concerns  also  arose  about  ease  of  use  and  the  amount  of 
concentration  required  to  read  in  this  way. 

TASK  6:  DESIGN,  BUILD,  &  EVALUATE  ON-HAND  ACCESS  APP:  -25%  COMPLETE 

Similar  to  Task  4,  the  progress  for  this  task  has  already  been  described  in  a  related  section  (Task 
3.1)  above.  To  summarize:  though  pervasive,  touchscreen  input  may  be  inaccessible  for  users  with 
visual  impairments,  or  in  certain  contexts  when  users  cannot  afford  visual  attention  (e.g.,  driving). 
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as  it  heavily  on  visual  feedback.  On-body  input,  which  is  performed  on  the  user’s  own  body,  could 
be  a  potential  solution  since  tactile  feedback  from  the  skin  enables  users  to  acquire  desired  target 
more  precisely  without  visual  feedback  [14],  For  example,  users  could  use  the  palm  of  their  hand 
as  a  substitute  for  a  mobile  phone’s  touchscreen,  performing  tap  and  swipe  gestures  to  control  the 
phone.  Alternatively,  particular  locations  could  be  tied  to  intuitive  commands,  such  as  tapping  on 
the  wrist  to  check  the  time  or  on  the  ear  to  answer  a  phone  call.  This  input  mechanism  may  be 
especially  beneficial  for  people  with  visual  impairments  [26,  35].  In  our  current  work  (which,  as 
previously  noted,  will  be  submitted  to  CVPR’16  in  early  November),  we  explore  computer  vision 
approaches  to  automatically  recognizing  regions  on  the  finger,  palm,  as  well  as  fingernails, 
knuckles,  wrist,  and  the  back-of-the-hand.  Using  a  dataset  consisting  of  30  individuals  and  17 
different  close-up  regions  on  the  hand,  our  algorithms  achieve  an  average  classification  accuracy  of 
96.4%.  However,  this  system  is  not  yet  wearable  and  our  algorithms  do  not  mn  in  real-time  (this 
remains  future  work). 


What  opportunities  for  training  and  professional  development  has  the  project  provided? 

If  the  project  was  not  intended  to  provide  training  and  professional  development  opportunities  or 
there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  opportunities  for  training  and  professional  development  provided  to  anyone  who 
worked  on  the  project  or  anyone  who  was  involved  in  the  activities  supported  by  the  project. 
“Training  ”  activities  are  those  in  which  individuals  with  advanced  professional  skills  and 
experience  assist  others  in  attaining  greater  proficiency.  Training  activities  may  include,  for 
example,  courses  or  one-on-one  work  with  a  mentor.  “Professional  development”  activities 
result  in  increased  knowledge  or  skill  in  one ’s  area  of  expertise  and  may  include  workshops, 
conferences,  seminars,  study  groups,  and  individual  study.  Include  participation  in  conferences, 
workshops,  and  seminars  not  listed  under  major  activities. 


As  university  professors,  we  work  closely  with  undergraduate  and  graduate  students  on  our  research 
projects.  HandSight  is  no  different.  Thus  far,  the  HandSight  project  has  involved  the  following  students 
(see  also  Section  7,  which  provides  a  more  detailed  list  of  personnel  and  roles;  asterisks  indicates 
funded  on  grant): 

PhD  Students:  Uran  Oh*,  Lee  Steams*,  Jonggi  Hong*,  Ruofei  Du 
Masters  Students:  Anis  Abboud* 

Undergraduate  Students:  Eric  Lancaster,  Bridget  Cheng,  Tony  Cheng,  Catherine  Jou,  Weishen 
‘Victor’  Chen,  Andrew  Matuza,  Yumeng  ‘Mandy’  Wang 

Our  PhD  students  lead  different  parts  of  the  project  and  gain  expertise  in  those  areas:  for  example,  Lee 
Steams  is  leading  the  computer  vision  component  of  the  work,  Jonggi  Hong  the  haptics  portion,  and 
Uran  Oh,  the  general  user  experience.  Our  PhD  students  also  help  supervise  undergraduate  students  in 
those  areas.  All  students  gain  both  technical  skills  in  their  areas  {e.g.,  computer  vision,  haptics),  skills 
related  to  designing  and  mnning  studies  {e.g.,  experimental  design,  analysis,  mnning  study  sessions 
themselves),  and  general  research  skills  {e.g.,  analyzing  data,  writing  papers).  Our  team  meets  once  a 
week  with  all  students  and  PIs/co-PIs.  David  Ross  joins  us  remotely  via  a  video  chat  interface.  Pis 
Froehlich  and  Findlater  also  conduct  weekly  1:1  meetings  with  the  graduate  students. 
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How  were  the  results  disseminated  to  communities  of  interest? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  results  were  disseminated  to  communities  of  interest.  Include  any  outreach 
activities  that  were  undertaken  to  reach  members  of  communities  who  are  not  usually  aware  of 
these  project  activities,  for  the  purpose  of  enhancing  public  understanding  and  increasing 
interest  in  learning  and  careers  in  science,  technology,  and  the  humanities. 


Our  team  attempts  to  disseminate  the  results  of  our  work  broadly  via  scientific  publications,  research 
talks  and  posters,  and  online  videos.  Thus  far,  we  have  presented  two  peer-reviewed  publications  at 
ACVR’  14  [43]  and  ASSETS’  15  [7],  We  also  have  three  papers  either  in  submission  or  nearing 
submission:  TACCESS,  CHI’ 16,  and  CVPR’16  (see  above  for  details). 

Pis  Froehlich  and  Findlater  often  include  HandSight  in  their  invited  talks.  For  example,  PI  Findlater 
was  recently  invited  to  give  a  talk  at  the  University  of  Illinois,  Urbana-Champaign  where  she  included 
content  on  HandSight. 

We  also  disseminate  our  work  beyond  the  academic  community  via  YouTube  videos,  webpages,  and 
tutorials.  For  example,  an  initial  HandSight  video  is  available  on  YouTube  here: 
https://voutu.be/AWCEDYdwBAc.  Moreover,  you  can  read  about  an  early  (pre-funded)  HandSight 
prototype  on  Instructables:  http://www.instructables.com/id/HandSight-A-Glove-for-the-Blind-to-Feel- 
Shapes-an/  where  it  has  been  viewed  over  21,000  times  and  favorited  107  times. 


Describe  briefly  what  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals 
and  objectives. 


TASK  1:  DEVELOP  RESPONSIVE  HAPTIC  FEEDBACK 

Our  haptic  feedback  work  has  been  driven  by  application  needs  and  we  expect  this  to  continue.  We 
have  pursued  two  simultaneous  haptic  feedback  options:  finger  mounted  and  wrist  mounted.  Our 
current  and  immediate  focus  is  on  the  wristband  haptics.  Our  CHI’ 16  work  (in  submission)  focuses 
purely  on  directional  guidance.  In  the  next  three  months,  we  plan  to  run  studies  of  target-finding  and 
route-tracing  tasks  along  with  co-PI  David  ross  at  the  Atlanta  VA.  Our  Android  testbed  is  fully 
complete,  we  have  just  been  finalizing  haptic  feedback  approaches  via  pilot  studies  before  running  a 
more  formal  study  at  the  Atlanta  VA. 

TASK  2:  DEVELOP  PHYSICAL  FORM  FACTOR: 

As  we  are  still  heavily  experimenting  with  hardware,  there  are  no  immediate  plans  to  further  develop 
our  physical  form  factor.  Instead,  our  focus  will  be  in  building  prototype  enclosures  and  wearable 
straps/hamesses  for  our  experimental  hardware.  This  will  indirectly  help  us  improve  the  functional 
aspects  of  future  form  factors 

TASK  3.  DEVELOP  AND  EVALUATE  SENSING  ALGORITHMS 

Real-Time  OCR  Algorithms.  Our  preliminary  algorithms  are  efficient  and  reasonably  accurate,  but 
there  is  much  room  for  improvement.  By  incorporating  constraints  on  lower-level  text  features  we  may 
be  able  to  rectify  vertical  perspective  effects  and  affine  skew.  We  can  also  apply  deblurring  and  image 
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stabilization  algorithms  to  improve  the  maximum  reading  speed  the  system  is  able  to  support.  Robust 
and  efficient  document  mosaicking  and  incorporation  of  prior  knowledge  will  likely  be  a  key 
component  for  supporting  a  wider  range  of  reading  tasks.  Currently,  our  prototype  relies  on  only  local 
information  gleaned  from  the  on-fmger  camera.  However,  in  the  future,  we  would  like  to  combine 
camera  streams  from  both  a  body-mounted  camera  (e.g.,  Orcam  [36])  and  a  finger-mounted  camera. 

We  expect  the  former  could  provide  more  global,  holistic  information  about  a  scene  or  text  which  could 
be  used  to  guide  the  finger  towards  a  target  of  interest  or  to  explore  the  physical  document’s  layout. 

On-Hand  Localization  Algorithms.  We  have  three  key  areas  of  future  work  here:  increasing 
robustness,  implementing  a  real-time  system,  and  reducing  training  requirements.  Towards  robustness, 
while  our  experiments  demonstrated  that  our  algorithms  are  largely  invariant  to  changes  in  rotation  and 
translation,  they  are  not  yet  invariant  to  scale,  lighting,  and  pose  (e.g.,  stretched  out  hand  vs.  folded 
hand).  A  self-illuminated  camera  that  is  fixed  in  place  on  the  user’s  finger  will  reduce  variations  in 
scale  and  lighting,  and  users  could  be  instructed  to  hold  their  hand  in  a  fixed  pose  while  using  the 
system.  However,  we  plan  to  also  investigate  ways  to  improve  our  algorithm’s  robustness  in  the  future, 
possibly  by  using  a  deformable  three-dimensional  model  of  the  hand  (e.g.,  [27,  28]).  Towards  a  real¬ 
time  system,  ultimately  we  want  to  optimize  our  algorithms  such  that  they  can  operate  on  a  modern 
smartwatch  (e.g..  Gear  S2).  Our  system  currently  operates  offline  on  an  Intel  i7  Desktop  system.  In  the 
next  year,  we  plan  to  optimize  and  adapt  our  algorithms  to  work  real-time  on  an  embedded  computer. 
Finally,  towards  reducing  training  requirements,  our  current  system  requires  between  5-10  training 
samples  per  hand  region.  This  could  be  perceived  as  overly  burdensome  by  some  users.  As  a 
comparison,  the  Apple  Fingerprint  Recognition  system  requires  7-10  samples  per  finger;  however, 
users  are  likely  to  only  train  1  or  2  fingers.  In  our  case,  its  likely  that  a  user  will  want  to  train  5-15 
separate  regions  on  their  hands  (if  not  more).  One  way  to  lower  training  requirements  is  to  find  useful 
cross-user  patterns  that  exist  across  all  hands/fmgers  that  can  be  used  to  bootstrap  the  system.  We  are 
currently  exploring  this  avenue. 

Color  and  Texture  Recognition.  While  we  had  one  undergraduate  student  (Gideon  Popkin) 
implement  some  preliminary  color  recognition  algorithms,  much  more  work  is  needed  here.  We  expect 
color  and  texture  recognition  will  be  a  key  part  of  Task  5:  Design,  Build,  and  Evaluate  a  Dressing 
Application,  which  our  SOW  outlines  for  Year  2. 

TASK  4:  DESIGN,  BUILD,  AND  EVALUATE  A  READING  APPLICATION 

Most  of  this  work  will  be  algorithmic  in  the  next  3-6  months  as  described  above.  We  plan  to 
experiment  with  a  multi-camera  approach  (e.g.,  two  instrumented  fingers  or  on-body  +  finger- 
mounted),  which  should  hopefully  improve  recognition  rates  and  solve  problems  due  to  limited  field- 
of-view,  blur,  etc. 

TASK  6:  DESIGN,  BUILD,  AND  EVALUATE  TECHNOLOGY  ACCESS  APPLICATION 

Once  we  have  created  a  real-time  on-hand  interaction  system,  we  plan  to  run  user  studies  with  VI  and 
control  participants  to  study  how  on-hand  interaction  compares  to  traditional  interactions  with  mobile 
devices. 
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4.  IMPACT:  Describe  distinctive  contributions,  major  accomplishments,  innovations,  successes,  or 
any  change  in  practice  or  behavior  that  has  come  about  as  a  result  of  the  project  relative  to: 

What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  findings,  results,  techniques  that  were  developed  or  extended,  or  other  products 
from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  the  base  of  knowledge, 
theory,  and  research  in  the  principal  disciplinary  field(s)  of  the  project.  Summarize  using 
language  that  an  intelligent  lay  audience  can  understand  (Scientific  American  style). 


Nothing  to  Report.  (See  also  Section  3  Accomplishments  above) 


What  was  the  impact  on  other  disciplines? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  findings,  results,  or  techniques  that  were  developed  or  improved,  or  other 
products  from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  other  disciplines. 


Nothing  to  Report 


What  was  the  impact  on  technology  transfer? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  ways  in  which  the  project  made  an  impact,  or  is  likely  to  make  an  impact,  on 
commercial  technology  or  public  use,  including: 

•  transfer  of  results  to  entities  in  government  or  industry; 

•  instances  where  the  research  has  led  to  the  initiation  of  a  start-up  company;  or 

•  adoption  of  new  practices. 


Nothing  to  Report 


What  was  the  impact  on  society  beyond  science  and  technology? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  results  from  the  project  made  an  impact,  or  are  likely  to  make  an  impact,  beyond 
the  bounds  of  science,  engineering,  and  the  academic  world  on  areas  such  as: 
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improving  public  knowledge,  attitudes,  skills,  and  abilities; 

changing  behavior,  practices,  decision  making,  policies  (including  regulatory  policies), 
or  social  actions;  or 

improving  social,  economic,  civic,  or  environmental  conditions. 


Nothing  to  Report 


5.  CHANGES/PROBLEMS:  The  Project  Director/Principal  Investigator  (PD/PI)  is  reminded  that 
the  recipient  organization  is  required  to  obtain  prior  written  approval  from  the  awarding  agency 
Grants  Officer  whenever  there  are  significant  changes  in  the  project  or  its  direction.  If  not 
previously  reported  in  writing,  provide  the  following  additional  information  or  state,  “Nothing  to 
Report,”  if  applicable: 


Changes  in  approach  and  reasons  for  change 

Describe  any  changes  in  approach  during  the  reporting  period  and  reasons  for  these  changes. 
Remember  that  significant  changes  in  objectives  and  scope  require  prior  approval  of  the  agency. 


Nothing  to  Report. 


Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

Describe  problems  or  delays  encountered  during  the  reporting  period  and  actions  or  plans  to 
resolve  them. 

No  major  problems. 


Changes  that  had  a  significant  impact  on  expenditures 

Describe  changes  during  the  reporting  period  that  may  have  had  a  significant  impact  on 
expenditures,  for  example,  delays  in  hiring  staff  or  favorable  developments  that  enable  meeting 
objectives  at  less  cost  than  anticipated. 


No  major  changes. 
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Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards, 
and/or  select  agents 

Describe  significant  deviations,  unexpected  outcomes,  or  changes  in  approved  protocols  for  the 
use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select  agents  during  the 
reporting  period.  If  required,  were  these  changes  approved  by  the  applicable  institution 
committee  (or  equivalent)  and  reported  to  the  agency?  Also  specify  the  applicable  Institutional 
Review  Board/Institutional  Animal  Care  and  Use  Committee  approval  dates. 

Significant  changes  in  use  or  care  of  human  subjects 

Nothing  to  Report 


Significant  changes  in  use  or  care  of  vertebrate  animals. 

Nothing  to  Report 


Significant  changes  in  use  of  biohazards  and/or  select  agents 


6.  PRODUCTS:  List  any  products  resulting  from  the  project  during  the  reporting  period.  If 
there  is  nothing  to  report  under  a  particular  item,  state  “Nothing  to  Report.” 


Publications,  conference  papers,  and  presentations 

Report  only  the  major  publication(s)  resulting  from  the  work  under  this  award. 

Journal  publications.  List  peer-reviewed  articles  or  papers  appearing  in  scientific, 
technical,  or  professional  journals.  Identify  for  each  publication:  Author(s);  title; 
journal;  volume:  year;  page  numbers;  status  of  publication  (published;  accepted, 
awaiting  publication;  submitted,  under  review;  other);  acknowledgement  of  federal 
support  (yes/no). 

In  Submission: 

1.  Steams,  L.,  Du,  R.,  Oh,  U.,  Jou,  C.,  Findlater,  L.,  Ross,  D.,  &  Froehlich,  J. 
Evaluating  Haptic  and  Auditory  Directional  Guidance  to  Assist  Blind  Persons 
in  Reading  Printed  Text  Using  Finger-Mounted  Cameras.  In  submission  to 
ACM  Transactions  on  Accessible  Computing. 
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Books  or  other  non-periodical,  one-time  publications.  Report  any  book,  monograph, 
dissertation,  abstract,  or  the  like  published  as  or  in  a  separate  publication,  rather  than  a 
periodical  or  series.  Include  any  significant  publication  in  the  proceedings  of  a  one-time 
conference  or  in  the  report  of  a  one-time  study,  commission,  or  the  like.  Identify  for  each 
one-time  publication:  Author(s);  title;  editor;  title  of  collection,  if  applicable; 
bibliographic  information;  year;  type  of  publication  (e.g.,  book,  thesis  or  dissertation); 
status  of  publication  (published;  accepted,  awaiting  publication;  submitted,  under 
review;  other);  acknowledgement  of  federal  support  (yes/no). 


None 


Other  publications,  conference  papers,  and  presentations.  Identify  any  other 
publications,  conference  papers  and/or  presentations  not  reported  above.  Specify  the 
status  of  the  publication  as  noted  above.  List  presentations  made  during  the  last  year 
(international,  national,  local  societies,  military  meetings,  etc.).  Use  an  asterisk  (*)  if 
presentation  produced  a  manuscript. 


Published: 

1 .  Stearns,  L.,  Du,  R.,  Oh,  U.,  Wang,  Y.,  Chellappa,  R.,  Findlater,  L.,  &  Froehlich,  J. 
E.  (2014).  The  Design  and  Preliminary  Evaluation  of  a  Einger-Mounted  Camera 
and  Eeedback  System  to  Enable  Reading  of  Printed  Text  for  the  Blind.  Workshop 
on  Assistive  Computer  Vision  and  Robotics  (ACVR’14)  in  Conjunction  with  the 
European  Conference  on  Computer  Vision  (ECCV'14).  doi:  10.1007/978-3-3 19- 
16199-0_43 

2.  Eindlater,  L.,  Steams,  L.,  Du,  R.,  Oh,  U.,  Ross,  D.,  Chellappa,  R.,  &  Eroehlich,  J. 
(2015).  Supporting  Everyday  Activities  for  Persons  with  Visual  Impairments 
Through  Computer  Vision- Augmented  Touch.  In  Proceedings  of  the  17th 
International  ACM  SIGACCESS  Conference  on  Computers  &  Accessibility  (pp. 
383-384).  New  York,  NY,  USA:  ACM.  doi:10.1 145/2700648.281 1381 

In  Submission: 

I.  Stearns, L.,  Oh, U.,  Cheng,  B.,  Chellappa, R., Eindlater, L.,  &  Eroehlich,  J.  E. 
(2016).  Contactless  Partial  Palmprint  Recognition:  Toward  a  Wearable  Einger- 
Camera  System  using  the  Hand  as  an  Input  Surface.  Will  be  submitted  to 
CVPR’16  in  early  November,  2015. 
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Website(s)  or  other  Internet  site(s) 

List  the  URL  for  any  Internet  site(s)  that  disseminates  the  results  of  the  research 
activities.  A  short  description  of  each  site  should  be  provided.  It  is  not  necessary  to 
include  the  publications  already  specified  above  in  this  section. 


•  http://www.leesteams.com/proiects.html  (student  run  project  website) 

•  http://www.cs.umd.edu/~iopf/publications.html  (PI  website) 

•  https://voutu.be/AWCEDYdwBAc  (ACVR’  14  HandSight  video) 

•  http://www.instructables.com/id/HandSight-A-Glove-for-the-Blind-to-Feel-Shapes-an/ 

(Instructable) 


Technologies  or  techniques 

Identify  technologies  or  techniques  that  resulted  from  the  research  activities.  In  addition 
to  a  description  of  the  technologies  or  techniques,  describe  how  they  will  be  shared. 


None. 


Inventions,  patent  applications,  and/or  licenses 

Identify  inventions,  patent  applications  with  date,  and/or  licenses  that  have  resulted  from 
the  research.  State  whether  an  application  is  provisional  or  non-provisional  and  indicate 
the  application  number.  Submission  of  this  information  as  part  of  an  interim  research 
performance  progress  report  is  not  a  substitute  for  any  other  invention  reporting 
required  under  the  terms  and  conditions  of  an  award. 


None 


Other  Products 

Identify  any  other  reportable  outcomes  that  were  developed  under  this  project. 
Reportable  outcomes  are  defined  as  a  research  result  that  is  or  relates  to  a  product, 
scientific  advance,  or  research  tool  that  makes  a  meaningful  contribution  toward  the 
understanding,  prevention,  diagnosis,  prognosis,  treatment,  and/or  rehabilitation  of  a 
disease,  injury  or  condition,  or  to  improve  the  quality  of  life.  Examples  include: 

•  data  or  databases; 

•  biospecimen  collections; 

•  audio  or  video  products; 

•  software; 
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•  models; 

•  educational  aids  or  curricula; 

•  instruments  or  equipment; 

•  research  material  (e.g.,  Germplasm;  cell  lines,  DNA  probes,  animal  models); 

•  clinical  interventions; 

•  new  business  creation;  and 

•  other. 


None 


7.  PARTICIPANTS  &  OTHER  COLLABORATING  ORGANIZATIONS 
What  individuals  have  worked  on  the  project? 

Provide  the  following  information  for:  (1)  PDs/PIs;  and  (2)  each  person  who  has  worked  at  least 
one  person  month  per  year  on  the  project  during  the  reporting  period,  regardless  of  the  source 
of  compensation  (a  person  month  equals  approximately  160  hours  of  effort).  If  information  is 
unchanged  from  a  previous  submission,  provide  the  name  only  and  indicate  “no  change.  ” 

Example: 


Name: 

Mary  Smith 

Project  Role: 

Graduate  Student 

Researcher  Identifier  (e.g.  ORCID 

ID):  1234567 

Nearest  person  month  worked: 

5 

Contribution  to  Project: 

Ms.  Smith  has  performed  work  in  the  area  of 
combined  error-control  and  constrained  coding. 

Funding  Support: 

The  Ford  Foundation  (Complete  only  if  the  funding 
support  is  provided  from  other  than  this  award). 

Name: 

Jon  Froehlich 

Project  Role: 

Pl/Professor 

Researcher  Identifier  (e.g.  orcidid): 

N/A 

Nearest  person  month  worked: 

3 

Contribution  to  Project: 

No  change 

Name: 

Leah  Findlater 

Project  Role: 

Pl/Professor 

Researcher  Identifier  (e.g.  orcidid): 

N/A 

Nearest  person  month  worked: 

3 

Contribution  to  Project: 

No  change 
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Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


David  Ross 
Co-PI/Professor 
N/A 
1 

Meets  ^weekly  with  team  via  video  hangout. 
Provides  feedback  on  project  and  on  drafts  of 
papers. 


Name: 

Project  Role: 

Researcher  Identifier  (e.g  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Rama  Chellappa 
Co-PI/Professor 
N/A 
I 

Provides  feedback  on  CV  related  topics  on  project 


Name: 

Project  Role: 

Researcher  Identifier  (e.g  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 

Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 

Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Lee  Stearns 
Graduate  Student 
N/A 
12 

No  change 
Uran  Oh 

Graduate  Student 

N/A 

9 

No  change 

Anis  Abboud 
Graduate  Student 
N/A 
I 

No  Change.  Graduated  in  May  2015. 


Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 

Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Jonggi  Hong 
Graduate  Student 
N/A 
6 

No  change 

Catherine  Jou 
Undergraduate  Student 
N/A 
I 

No  change.  Graduated  in  May  2015. 
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Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 

Name: 

Project  Role: 

Researcher  Identifier  (e.g  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 

Name: 

Project  Role: 

Researcher  Identifier  (e.g  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Tony  Cheng 
Undergraduate  Student 
N/A 
3 

No  change. 

Weis  hen  ‘Victor’  Chen 
Undergraduate  Student 
N/A 
I 

No  change.  Graduated  in  May  2015. 

Andrew  Matuza 
Undergraduate  Student 
N/A 
I 

No  change.  Graduated  in  May  2015. 


Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Ruofei  Du 
Graduate  Student 
N/A 
2 

No  change.  Stopped  working  on  project  after  Q2. 


Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Yumeng  “Mandy”  Wang 
Undergraduate  Student 
N/A 
2 

No  change.  Stopped  working  on  project  after  Ql. 


Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Bridget  Cheng 
Undergraduate  Student 
N/A 
3 

Visiting  undergraduate  student  from  Cornell. 
Worked  with  Lee  Stearns  on  CVPR  data  collection 


Name: 

Project  Role: 

Researcher  Identifier  (e.g.  orcidid): 
Nearest  person  month  worked: 
Contribution  to  Project: 


Eric  Lancaster 
Undergraduate  Student 
N/A 
I 

Working  with  Lee  Stearns  on  follow-up  work  to 
CVPR  paper  using  IR  to  gather  sub-skin  data 
(e.g.,  veins)  to  improve  on-hand  localization 
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Has  there  been  a  change  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key  personnel 
since  the  last  reporting  period? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

If  the  active  support  has  changed  for  the  PD/PI(s)  or  senior/key  personnel,  then  describe  what 
the  change  has  been.  Changes  may  occur,  for  example,  if  a  previously  active  grant  has  closed 
and/or  if  a  previously  pending  grant  is  now  active.  Annotate  this  information  so  it  is  clear  what 
has  changed  from  the  previous  submission.  Submission  of  other  support  information  is  not 
necessary  for  pending  changes  or  for  changes  in  the  level  of  effort  for  active  support  reported 
previously.  The  awarding  agency  may  require  prior  written  approval  if  a  change  in  active  other 
support  significantly  impacts  the  effort  on  the  project  that  is  the  subject  of  the  project  report. 


Nothing  to  Report. 


What  other  organizations  were  involved  as  partners? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  partner  organizations  -  academic  institutions,  other  nonprofits,  industrial  or 
commercial  firms,  state  or  local  governments,  schools  or  school  systems,  or  other  organizations 
(foreign  or  domestic)  -  that  were  involved  with  the  project.  Partner  organizations  may  have 
provided  financial  or  in-kind  support,  supplied  facilities  or  equipment,  collaborated  in  the 
research,  exchanged  personnel,  or  otherwise  contributed. 

Provide  the  following  information  for  each  partnership: 

Organization  Name: 

Location  of  Organization:  (if  foreign  location  list  country) 

Partner’s  contribution  to  the  project  (identify  one  or  more) 

•  Financial  support; 

•  In-kind  support  (e.g.,  partner  makes  software,  computers,  equipment,  etc., 
available  to  project  stajf); 

•  Facilities  (e.g.,  project  staff  use  the  partner ’s  facilities  for  project  activities); 

•  Collaboration  (e.g.,  partner ’s  staff  work  with  project  staff  on  the  project); 

•  Personnel  exchanges  (e.g.,  project  staff  and/or  partner ’s  staff  use  each  other ’s  facilities, 
work  at  each  other ’s  site);  and 

•  Other. 


Nothing  to  Report. 


27 


8.  SPECIAL  REPORTING  REQUIREMENTS 

None  to  report 


9.  APPENDICES: 
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